Papers by Pham Khanh Chi
MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for reducing the computational cost of large language models (LLMs) focus on minimizing the divergence between the output probability distributions of the teacher and the student, which limits knowledge transfer. |
| Approach: | They propose a framework that aligns teacher and student representations along their layer-wise transformation trajectory. |
| Outcome: | The proposed framework outperforms state-of-the-art benchmarks on teacher–student layers. |
SRA: Span Representation Alignment for Large Language Model Distillation (2026.acl-long)
Copied to clipboard
Quoc Phong Dao, Hoang Son Nguyen, Pham Khanh Chi, Tung Nguyen, Linh Ngo Van, Nguyen Thi Ngoc Diep, Trung Le
| Challenge: | Existing approaches to cross-tokenizer knowledge distillation are brittle and sensitive to discrepancies between tokenizers. |
| Approach: | They propose a framework that shifts the unit of alignment from tokens to robust, tokenizer-agnostic spans and employ a geometric regularizer to preserve the structural integrity of the representation space. |
| Outcome: | The proposed framework outperforms state-of-the-art methods in cross-architecture distillation experiments. |
TALAS: Teacher-Anchored Layer Alignment with Adaptive Sharpness-Aware Minimization for Embedding Distillation (2026.acl-long)
Copied to clipboard
Quoc Phong Dao, Hoang Son Nguyen, Pham Khanh Chi, Linh Ngo Van, Nguyen Thi Ngoc Diep, Thien Huu Nguyen, Trung Le
| Challenge: | Existing methods that require a student to strictly mimic the teacher’s sentence embeddings or internal features often incur prohibitive computational costs and yield suboptimal performance due to the inherent capacity gap. |
| Approach: | They propose a Teacher-Anchored mechanism that selectively distills final sentence embeddings only into the student’s upper layers, thereby reducing overhead while respecting capacity constraints. |
| Outcome: | Empirical results show that TALAS outperforms existing methods while maintaining high performance. |
DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing cross-tokenizer distillation methods are limited by suboptimal alignment across sequence and vocabulary levels. |
| Approach: | They propose a cross-tokenizer distillation framework that enhances token-wise distillation . they use dual-space entropy-based weighting to achieve precise sequence-level alignment . |
| Outcome: | The proposed framework outperforms state-of-the-art methods in large language models but has high computational and memory costs. |